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Abstract 


Online video lectures have been considered an instructional media for various pedagogic 
approaches, such as the flipped classroom and open online courses. In comparison to other 
instructional media, online video affords the opportunity for recording student clickstream 
patterns within a video lecture. Video analytics within lecture videos may provide insights into 
student learning performance and inform the improvement of video-assisted teaching tactics. 
Nevertheless, video analytics are not accessible to learning stakeholders, such as researchers and 
educators, mainly because online video platforms do not broadly share the interactions of the 
users with their systems. For this purpose, we have designed an open-access video analytics 
system for use in a video-assisted course. In this paper, we present a longitudinal study, which 
provides valuable insights through the lens of the collected video analytics. In particular, we 
found that there is a relationship between video navigation (repeated views) and the level of 
cognition/thinking required for a specific video segment. Our results indicated that learning 
performance progress was slightly improved and stabilized after the third week of the video- 
assisted course. We also found that attitudes regarding easiness, usability, usefulness, and 
acceptance of this type of course remained at the same levels throughout the course. Finally, we 
triangulate analytics from diverse sources, discuss them, and provide the lessons learned for 
further development and refinement of video-assisted courses and practices. 

Keywords: User interactions; learning analytics; video lecture; open learning system; analytics 
triangulation; controlled experiment 
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Introduction 

The use of video for learning has become widely employed in recent years. Video-based learning 
techniques and practices are applied in a variety of ways, such as the flipped classroom, small 
private online courses (SPOCs), and xMOOCs. Learners enjoy video streaming from different 
platforms (e.g., YouTube) on a diverse number of terminals (TV, desktop, smart phone, tablets) 
and create billions of simple interactions. Observations of this learning activity might be 
converted via analytics into useful information (Giannakos et al., 2013) for the benefit of all video 
learners. Video learning analytics (LA) can allow researchers and educators to understand and 
improve the effectiveness of video-based learning tools and practices. 

It is essential that students are actively involved in the learning process; however, content is still 
often taught via the traditional lecture approach where students are placed in the role of passive 
learners, which involves limited options for interactivity and engagement with the course 
material. Prominent technologies that have gained much attention include social media such as 
wikis, Facebook, and YouTube (Siemens, 2011) for information exchange, collaboration, 
instruction, and interaction with the course materials. With the myriad of technologies available 
today, many instructors face the dilemma of how we can take full advantage of the instructional 
media to provide more effective student-centred instruction. 

Traditional lectures may no longer primarily serve the purpose of disseminating information, 
which can be easily retrieved from many online video lecture repositories at any time. Video 
lectures have given rise to flipped (or inverted) classrooms and even assist SPOCs (Fox, 2013). 
This specific type of blended-learning classroom utilizes technology, such as video, to move 
lectures outside the classroom, thereby giving students and teachers time for active learning in 
the classroom (Roehl, 2012). By using learning materials as a supplement to classroom teaching 
rather than being viewed as a replacement for it, those techniques are attempting to increase 
instructor leverage, student throughput, student mastery, and engagement (Fox, 2013). At the 
same time, recent technical and infrastructural developments (Giannakos, 2013) make the 
potential of video-assisted learning ripe for exploration. Capturing and sharing learners’ diverse 
interactions with the emerging learning technologies can clearly provide scholars and educators 
with valuable information. 

Video resources have emerged as one of the premier forms of learning materials. In our paper we 
use the term video-assisted learning to refer to the systematic use of video resources for the 
purpose of achieving defined competences. Hence, video-assisted learning might be defined as, 
the learning process of acquiring defined knowledge, competence, and skills with the systematic 
assistance of video resources. With the widespread adoption of online video lecture communities, 
such as Khan Academy and VideoLectures.net, conducting research to understand how students 
learn via video lectures has become critical. Despite the present significant body of related 
research into the impact of video lectures (Giannakos, 2013), the majority of previous efforts have 
mainly focused on: a sporadic or single use of video lectures in an educational context (Evans, 
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2008) and/or the investigation of only one factor like student performance (Kazlauskas and 
Robinson, 2012). Therefore, the longitudinal collection of diverse LA and the interpretation of 
them through triangulation will allow us to better understand how students learn and interact 
with videos. 

Researchers have recently begun to study student interactions with video-assisted learning in 
order to provide educators with valuable information about students (e.g., Khan Academy, 
Coursera). The capture and analysis of this information is still in an embryotic research stage, and 
the experimental instrumentation and methodology described herein will significantly enhance 
this critical research effort. 

The next section outlines the related work and the focus of this research; the third section 
presents the system, its validation, and its final adjustments; the fourth section describes the 
methodology of the study employed in this article; the fifth section exhibits the empirical results 
derived; and the last section discusses the results and limitations, suggests the implications, and 
makes recommendations for future research. 


Related Work 

Video lecture has emerged as one of the premier open educational resources (McGreal et al., 
2012). Today, advanced video repository systems have seen enormous growth (e.g. Khan 
Academy, PBS Teachers, Moma’s Modern Teachers) through social software tools and the 
possibilities to enhance videos on them. Most of the social software tools, including wikis, 
weblogs, Facebook, Twitter, MySpace, and e-portfolios, can potentially provide a vehicle to 
promote video lectures. 

In addition, many instructors in higher education are implementing video lectures in a variety of 
ways, such as broadcasting lectures in distance education (Maag, 2006), delivering recordings of 
in-class lectures with face-to-face meetings for review purposes (Brotherton and Abowd, 2004), 
and delivering lecture recordings before class to conserve class time and thereby flipping the day 
for hands-on activities (Day and Foley, 2006). Other uses include showing videos that 
demonstrate course topics (Jadin et al., 2009) and providing supplementary video learning 
materials for self-study (Dhonau and McAlpine, 2002). Researchers have delineated the 
educational advantages and disadvantages of video lectures (Traphagan et al., 2010; Ljubojevic et 
al., 2014). However, previous efforts have mainly focused on the sporadic use of video lectures 
and the investigation of a specific feature. 

Students using video lectures enjoy control over when and where they learn, what they need to 
learn, and the pace of their learning (Heilesen, 2010). The study habits of those students using 
video lectures have been shown to improve, including a fostering of independence (Jarvis and 
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Dickie, 2009), an increase in self-reflection (Leijen et al., 2008), the heightening of efficient test 
preparation (McCombs and Liu, 2007), and the practice of reviewing material more regularly 
(O’Bryan and Hegelheimer, 2007). Learner control in well-designed video lectures can be 
beneficial in terms of convenience and supplemental practice (Hannafin, 1984). Students use 
videos with many different patterns (Ullrich et al., 2013) and report a variety of reasons for using 
video lectures (Traphagan et al., 2010; Donkor, 2011). Van Zanten et al. (2012) indicate that 
students widely use video lectures for revision and review purposes during exam preparation. 
Harris and Park (2008) also argue that video lectures can also be used for several reasons, 
including dissemination of material, supplementing class materials, guest lecture presentations, 
and even as a marketing tool for attracting prospective customers and students. 

When video lectures are available, students typically use them. For instance, Harley et al. (2003) 
found that almost all students (95-97%) viewed video lectures at least once. Our motivation for 
this work is based on the following issues. First, the use of videos for learning has become widely 
utilized in recent years (Giannakos, 2013); video-based technological tools have been developed, 
and many educational institutions and digital libraries have incorporated video into their 
instructional media portfolio. Second, despite the growing number and variety of video lectures 
available, there is limited understanding of the nature and quality of their effectiveness, in terms 
of how students use and learn from video lectures. Specifically, limited research currently exists 
regarding guidelines for the use of video lectures and the design of pedagogical systems for their 
use. 

Innovative features, such as slide-video separation, social categorization and navigation, and 
advanced search, have become standards for any state of-the-art system. Working with video 
lectures has essential differences from working with traditional or even digital textbooks. Video 
lectures are easy to watch at a normal pace with extra affordances like fast-forward and rewind 
serving as useful tools for learners (Giannakos and Vlamos, 2013). While video lectures lack 
textbooks’ font and typography that allow learners and instructors to emphasize key points and 
categorize the content in many ways, they can provide extra information using video-pace, voice 
tone, expression of emotions, visual cues, and many other forms of social information. 

Other innovative social features like collaborative video annotation and social navigation have 
also been used recently in video learning platforms (Risko et al., 2013; Wilk et al., 2013; Torres- 
Ramirez et al., 2014). Collaborative video annotation allows learners to comment and discuss 
specific points of the video lecture, whereas social navigation engages crowdsourcing to collate 
the whole-class watching behavior, showing which fragments were watched most frequently 
(Risko et al., 2013). As the number of students studying the same topic grows and our ability to 
capture their learning patterns, we can expect even more exciting socially driven technologies to 
be explored and brought into our everyday life. 

In summary, students are using video lectures for a variety of subjective and objective benefits, 
and students perceive video technology as a practical learning resource. However, many aspects 
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related with a) students’ video-navigation, b) learning performance, and c) attitudes with the 
video-assisted learning still remain unexplored: 

a) Navigation - Are students viewing the entire video lecture?; what segments of the video 
lecture do students select to view, and why?; how many times do students view any given 
video lecture?; and what video applications are more attractive or engaging? 

b) Learning performance - How do students perform with the assistance of video 
materials?; is there any relation among students’ viewing and learning performance? 

c) Attitudes - How do students perceive video-assisted learning?; is there any significant 
shift in their attitudes during a video-assisted course? 

To address these critical issues, this study provides a first step towards understanding students’ 
multi-faceted interactions with video lectures. In particular, this study was designed to assess and 
make sense of the analytics within video lectures and to investigate the relationship between these 
analytics and students’ attitudes and learning performance. To do so, we designed and deployed a 
longitudinal (7—week) study based on our video learning analytics system. Using this system, we 
collected and analyzed students’: 1) video navigation, 2) learning performance, and 3) attitudes; 
based on these diverse sources, learning analytics, and the respective interpretation through data 
triangulation, we provide the new information for further development and refinement of video- 
assisted courses and practices. 


Building and Validating the Video Learning Analytics 

System 

In this section, we present the development and validation of the video learning analytics system 
(VLAS). The VLAS consists of: (1) YouTube Application Programming Interface (API), (2) Google 
App Engine (GAE), and (3) Eclipse (Java), all of which were seamlessly integrated into a flexible 
architecture (Figure 1). The selected tools (GAE, YouTube, Google accounts) offer multiple 
benefits. GAE enables the development of web-based applications, as well as maintenance and 
administration of the traffic and the data storage. YouTube allows developers to use its 
infrastructures (e.g., YouTube videos) and provides a chrome-less user interface, which is a 
YouTube video player without any controls, thereby facilitating customization within Flash or 
HTML 5. As a result, we used JavaScript to create custom buttons and to implement their 
functions. 
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Interaction 
buttons and 
questionnaires 
along with the 
connections of 
the system to 
be developed 


Figure l. The architecture of the Video Learning Analytics System. 


In order to test, validate, and finally improve the system, we conducted an empirical system 
validation study. The main goal of the user experiment was to collect some first activity data from 
the learners and to establish a flexible experimental procedure that can be replicated and 
validated by other researchers. Instead of mining real usage data, we designed a controlled 
experiment to provide a clean set of data. The experiment took place in a lab with Internet 
connections, general-purpose computers, and headphones. Twenty-three university students (18- 
35 years old, 13 females and 10 males) spent approximately ten minutes watching each video. All 
students had been attending the Human-Computer Interaction courses at the Department of 
Informatics at a post- or under-graduate level and received course credit in the respective courses. 
Next, a time restriction of five minutes was imposed, in order to motivate the users to actively 
interact with the video analytics system by browsing through the video and answering the 
respective questions. We enabled the Replay30 and Skip30 buttons, and we informed the users 
that the purpose of the study was to measure their performance in finding the answers to the 
questions within time constraints. Although further research could progress to larger scale 
studies, the observable connection between the learners’ behavior data and the important 
segments detection affirms the system via a clearly replicable process and should provide 
assurance of the validity of the learning analytics tool. 

The questionnaire consisted of very simple questions that could not be answered through 
previous knowledge of the users. For instance, “Which are the main research topics?”; “What is 
the purpose of hackers?”; and “Which is the right order for mixing the ingredients?”. We selected 
three types of common videos at different levels (research, elementary, technical) of online video 
learning: 1) university lecture, 2) documentary, and 3) how-to. 
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In order to present the results of our field study, we used graphs that facilitated the visual 
comparison between l) the original learner activity segments (LAS), 2) the rich information 
segments (RIS), and 3) smooth versions of the LAS (Figure 2, up). Next, we visually compared the 
smooth versions of the component and composite times series to the RIS (Figure 2, down). We 
observed that in most cases the Replay30 time series closely matched the RIS. Neither the Skip30 
nor the composite time series seem to match the RIS (Figure 2, down). 



LAS 

LAS (Smooth 30) 
RIS 

LAS (Smooth 60) 
LAS (Smooth 10) 



Figure 2 . The original learner activity segments (LAS), the rich information segments (RIS) and 
smooth versions of the LAS (up); The RIS, the Replay30 based LAS, the Skip30 based LAS and 
the composite (Skip30-Replay30) based LAS. 


Therefore, we computed the local maximums of the Replay30 time series for each one of the three 
videos (see Table l). 
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Table l 

The Distance of the Local Maximum of the Replay 30 Time Series from the Start of the Respective 
Pulse (Inside Parenthesis) in the RIS Time Series 


RIS 

Video A 

Video B 

Video C 

Distance of local Replay30 max (from RIS start) 

RIS1 

33 (40) 

58 (1) 

45 (105) 

RIS2 

13 (145) 

19 (126) 

21 (230) 

R1S3 

48 (350) 

16 (229) 

-13 (374) 

R1S4 

1 (554) 

15 (338) 

21 (475) 


Based on the user experiment, we improved the system and transformed it into an online service 
( http://www.socialskip.org/ : see also Chorianopoulos et al. 2014). Users visit the link rather than 
going through an installation process. If there is an updated version, they simply have to refresh 
the page. Additionally, the system’s architecture is modular and allows re-use of the components. 
Web developers might employ the open-source application logic 
(https://code.google.eom/p/socialskip/) in order to develop new or custom features. 

Practically every user with a Google account can be a researcher. To do so, one must merely sign 
in to the service, connect the selected video from YouTube, configure the video player buttons or 
slider, and copy-paste the address of the online survey he/she wants to use (see Figure 3, up). By 
taking these few steps, a respective video assessment (or experiment) URL is created (Figure 3, 
down). 
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The researcher adds 
a short description 


The researcher copies the URL of the 
respective YouTube video 


Description: 

VideolIRL: 


Buttons: o 
Slider: • 


Play/Pause: a 
Forward; ■ 
Backward: bi 
J ump: sec 

Interaction: min 


Options for 
configuring the 
buttons of the 
experiment 


Questionnaire: 

Information: 




Information prior to 



The researcher copies 


At the end an 

the experiment can be 



the URL of the 


experiment 

provided to the user 


respective web-survey 

—► 

URL is created 


Video Experiment url 


I soda Iskip.a ppspot.com/?videold= 7601 


Figure 3. Experiment creation window (up); video experiment url (down). 


The instructor/researcher can then share this URL with his students or post it in course’s 
blog/wiki webpage. Hence, this video analytics system simplifies and makes feasible for everyone 
to incorporate video assessments on his/her syllabus. This URL contains the learner/user- 
interface of the video analytics system (Figure 4). The video analytics system employs the utilities 
selected by the researcher buttons. 



Figure 4. The interface of the system has familiar buttons, seek bar, as well as questionnaire 
functionality. 
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Learners have the option of using a personal Google account to sign in and watch the uploaded 
videos. In this way, we can accomplish user authentication and avoid the necessity of 
implementing a user account system specifically for the application. Thus, users’ interactions are 
recorded and stored according to their Gmail addresses. Each time a user signs into the web-video 
player application, a new log is created. Whenever a button is pressed, an abbreviation of the 
button’s name and the time it occurred are stored. 

When the researcher terminates the specific experiment, he can visit the 
configuration/management area of his experiment to l) download all the collected data, 2) 
visualize the activity of each video, 3) configure the experiment, and 4) delete it thereafter (Figure 
5). These options give the researcher the flexibility to test different activities or functionalities on 
different groups of students, analyze the results, and develop useful conclusions about how 
students use and learn from video-based learning systems. 



Figure 5. The interface of researchers configuration/management area. 


Concerning the visualization capabilities of the system, we have opted to use time series to 
represent learner activity. A time series is a sequence of data points, measured typically at 
successive points in time and spaced at uniform time intervals. Time series analysis provides 
methods for analyzing time series data to extract meaningful statistics and other characteristics of 
the data (Hamilton, 1994). Figure 6 illustrates an example of the visualization of the learner 
activity via the time series technique. Time series graph depicts user interactions with the video 
lecture. In addition to the preselected visualizations of time series, the user of the system has the 
option to download the data-set locally in a comma separated values (CSV) format. Then, the 
researcher might import the CSV file into a visualization program of preference (e.g., R) for more 
advanced analysis and graphics. 
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How much important is the video each time moment (0 = low, 1 = very) 



Time (sec) 

Figure 6. Exemplar visualization of learners’ viewing activity. 


Methodology 


Sampling and Procedures 

In our effort to investigate students’ viewing patterns and interests, we conducted a small-scale 
longitudinal study. The sample of the study consisted of science majors who selected to enroll in 
an undergraduate reading course. The instructors developed a video lecture syllabus to assist 
students’ before-class preparation. The video lectures featured native speakers, and the pace and 
style of all the videos was the same. The video lectures were integrated with the appropriate 
assessments, consisting of questions that could be answered based on the content of the video. 
The respective video-lectures (with the integrated assessments) were posted to the wiki of the 
course for before-class use. Eleven freshmen (18-20 years old, 6 females and 5 males) participated 
in the study. The course has well-defined learning objectives and content, and the study took 
place at a public university. The learning objectives of the course were related to the use of 
information for problem solving, research, and decision making. The course lasted approximately 
10 weeks, enhanced with video lecturers to assist students during a 7-week period (the first 2 
weeks and the last one did not require video lectures). For the distribution and management of 
the video lectures, we employed our video learning analytics system. 

Research Design 

The research design of our study is a single-group time series design (Ross and Morrison, 1996), 
involving repeated measurement of a group with the experimental treatment induced between 
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two of the measures. The single-group time series design can be diagrammed as shown below (l). 
As depicted, one group (G) is observed (O) and receives the treatment (X) several times. 

G Oi Xi 02 X2 O3 (1) 

Our study consists of seven repeated treatments and seven measures of the video analytics 
(navigation) and learning performance, as well as pre-post measurement of students’ attitudes. 

Measures 

We incorporated a respective assessment for each video lecture, consisting of questions that could 
be answered based on the content of the video lecture. The video-assessment integration provided 
students with higher cognitive level learning and urged them to navigate and achieve deeper 
engagement with the video lecture. 

In addition to the data collected via the assessment and the video learning analytics system 
(students’ navigation), we integrated a short questionnaire at the first (pre) and the last (post) 
video assessment. The questionnaire included measures of 1) ease to use, 2) control, 3) intention 
to use, and 4) usefulness of the video-assisted course. Table 2 lists the operational definitions and 
the number of items (questions) of each of the constructs (measures), as well as the source from 
which the measures were adopted. We employed a 7-point Likert scale anchored from 2 
(“completely disagree”) to 7 (“completely agree”). 

Table 2 

The Attitudinal Measures, Its Definitions, And The Number Of Items Used 


Construct 

Definition 

# of items 

Source 

Easy to Use 

The degree to which an individual believes that 
participating in a video-assisted course is easy 
for him/her. 

3 

(Ngai 
et al., 

07 ) 

Control 

The degree to which a person perceives how 
easy or difficult it would be to perform an 
operation during a video-assisted course. 

2 

(Shih, 

08) 

Usefulness 

The degree to which an individual believes that 
participating in a video-assisted course is 
useful for him/her. 

3 

(Ngai 
et al., 

07 ) 

Intention to 

Participate 

The degree of students’ intention to participate 
in similar video-assisted courses in the future. 

3 

(Lee et 
al., 09) 
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In summary, the data collection of our study can be divided in three basic categories: 

a) Students’ video navigation (collected via video learning analytics system), 

b) Students’ learning performance/score (collected via the integrated with the system 
assessments) and, 

c) Students’ attitudes toward the video-assisted course (collected via the integrated with the 
system questionnaires). 

Data Analysis 

As aforementioned, the collected data consists of three different types; therefore, an appropriate 
data analysis was used for each different set of data. Per students’ video navigation, we used the 
aggregated time series visualization, in order to identify the peaks of students’ video views (global 
maximums). Afterwards, we investigated any potential relation between students’ views and 
learning performance in the respective video-based assessments. At the end of our analysis, we 
watched the respective global peak of each video, and we attempt to give an explanation of why 
those segments are so important for students. 

Regarding students’ learning performance, we captured all students’ assessments scores and 
mapped them in a week-by-week diagram. In this way we are able to address students’ progress 
throughout the video-assisted course. In addition, we divided video assignments in those with low 
and high scores (based on students’ learning performance scores); afterwards we tried to 
understand any differences in students’ video navigation. 

In order to identify any potential shift in students’ attitudes during the video-assisted course, we 
used t-tests between the pre- and the post-results of their attitudes as exhibited from the 
questionnaires. Hence, we were able not only to capture students’ attitudes toward the video- 
assisted course, but also to identify any potential shift during the course. 


Research Findings 

With the visualization of the students’ activity using graphs, we reach the conclusion that there is 
a positive relation among students’ learning performance scores and their views on the respective 
video assessment. As we can see from Figure 7, in videos where students’ exhibited low score they 
had less repeated views as exhibited from the low level of importance, which was <0.5 most of the 
time. On the other hand, in videos where students exhibited high scores (e.g. Fig. 7, right) they 
had many repeated views, as exhibited from the high level of importance, which was almost 
always > 0.5. Hence, we can indicate that video lecture production, which results in different 
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video navigations, affects students’ learning performance; and that “attractive” videos result in 
better learning outcomes. 

Trying to identify the “attractive” (many views) video segments, we found that activity peaks 
(global-local maximums) were identified at the video segments with information related to the 
answers of the assessment and at the segments where the presenter was giving the solution of the 
respective problem. So the main quality of the “attractive” video segments was the rich and useful 
amount of transferred information and knowledge. 

How much important is the video each time moment (0 = low, 1 = very) 



Time (sec) 


Low Score Video High Score Video 

Figure 7. The visualization low (left) and high (right) importance videos, based on students’ 
views. 


However, it was still unclear why students found some video segments extremely important 
(global maximums). In order to be able to provide an explanation about this issue, we went 
through the seven video activity graphs (seven video lectures were offered) and located the global 
maximums (see Figure 8). We then reviewed those segments in correspondence with the rest of 
the video and the local maximums. 
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Figure 8. Students’ activity graphs, identifying the extremely important (global maximums) and 
unimportant (global minimums) video segment. 


By doing this explorative investigation, we reach the conclusion that there is a correspondence 
between the level of cognition/thinking each question required and the size of the respective peak. 
Using the revised taxonomy of Bloom (see Table 3) (Anderson et al., 2001) we perceived that all 
the global maximums were identified in questions where higher order thinking/cognitive skills 
was required. This can be explained by the fact that similar studying behavior occurs with 
traditional or even digital textbooks, when students re-read text corpus where higher order 
thinking/cognitive skills are required. 
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Table 3 

Bloom's Taxonomy Stages and Action World Examples, as Revised by Anderson et al. in 2001 


Level of eogn itie n/thinking 

Action words 

Remembering: can the student 
recall or remember the 

information? 

Define, duplicate, list memorize, recall, 
repeat, reproduce, state 

Understanding: can the student 
explain ideas or concepts? 

Describe, discuss, explain, identify, locate, 
recognize, repost, select, translate, 
paraphrase 

Applying: can the student use the 
information in a new way? 

Choose, demonstrate, dramatize, employ, 
illustrate, interpret, operate, schedule, 
sketch, solve, use, write 

Analyzing: can the student 
distinguish between the different 
parts? 

Compare, contrast, criticize, discriminate, 
distinguish, examine, experiment, 

question, test 

Evaluating : can the student justify 
a stand or decision? 

^Appraise, argue, defend, judge, select, 
support, value, evaluate 

Creating: can the student create a 
new product of point of view? 

Assemble, construct, create, design, 
develop, formulate, write 


At the end of the course, we collected students’ assessment scores (Figure 9) and noticed that 
students had the lowest score on the first assessment and the highest score on the last one. In 
addition, the last assessment has the smaller standard deviation value. Based on the results, we 
can indicate that after the first three video assessments, students exhibited better and more 
robust (lower standard deviation) scores. Given that all the video assessments had the same 
difficulty since the answers were always within the video and were presented in a similar way, we 
can assume that the video assisted process became more familiar after the third week and 
students adjusted their studying accordingly. In addition, the fact that the standard deviations 
minimized supports the notion that low performers benefited more, which is also in alignment 
with instructors’ notes and comments. 
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Figure 9. Students’ week by week video assessments scores. 


Regarding students’ attitudes toward the video-assisted course, we used an attitudinal 
questionnaire to assess them (Additionally Table 2). Students expressed high Control and 
Usefulness (6.5/7) during the video enhancement., they expressed a slightly lower (though still 
very high) level on the Easy to Use (6.27/7) quality of the video enhancement and their Intention 
to Use (6.35/7) h in the future. High levels of these constructs indicate positive views concerning 
usability, control, and usefulness regarding the video-assisted course. 

To examine any potential shift in students’ attitudes during the video-assisted course, we used t- 
tests between the pre- and the post-scores of the attitudinal questionnaires. As we can see from 
Figure 10, students’ responses were at very high levels in both the pre and the post 
questionnaires. Performing a t-test between the pre- and post-scores of the questionnaire, the 
results showed no significant difference among all of our constructs: Easy to Use t(2o) = 0.59, p > 
.05; Control t(2o) = -0.36, p > .05; Intention to Use t(2o) = 0.18, p > .05; Usefulness t(2o) = 1.09, 
p > .05. As a consequence, there was no shift in students’ attitudes after the 7-week period; 
therefore, it can be concluded that the video-assisted course is considered stably useful, usable, 
and well-received by the students. 


■ Pre-Score ■ Post-Score 



Easy to Use Control Intention to Use Usefulness 


Figure 10. Students’ attitudes in the pre- and post-questionnaires. 
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Discussion and Conclusions 

Millions of learners enjoy video streaming from different platforms (Coursera, Khan Academy, 
EdX, Udacity, Iversity, Futurelearn), creating billions of simple interactions. This data might be 
converted into useful information for the benefit of all video learners. As the practice of learning 
through videos on web-based systems increases, more and more interactions are going to be 
gathered. Dynamic analysis of this wealth of data will allow us to better understand learners’ 
experience. In addition, the combination of richer user profiles and content metadata will provide 
opportunities for adding value to data obtained from video-assisted learning. 

Although many corporations and academic institutions are making lecture videos and seminars 
available online, there have been few and scattered research efforts (i.e., Traphagan et al., 2010) 
to understand and leverage actual learner experience. Moreover, Kim et al. (2014) have provided 
analytics for millions of interactions, which have been produced by thousands of MOOC students. 
Nevertheless, video analytics in controlled conditions provide more than massive amounts of 
video interaction data, allowing us to make more interpretations, between-group comparisons, 
and investigations into hybrid learning settings. To the best of our knowledge, there are currently 
no efforts triangulating diverse data, such as interactions with the system and students’ 
performance and attitudes, in order to derive valuable information about how students use and 
ultimately learn via video systems. In addition, video materials can be used in blended settings, 
SPOCs, and flipped classrooms to supplement and improve the classroom experience. Many 
flipped classroom and SPOCs learning processes and experiments will be deployed in the near 
future, some of them will probably succeed and others will fail. Regardless of each future success 
or failure, the sensemaking, pluralism, and triangulation of the collected analytics will allow us to 
provide a more precise and realistic viewpoint from each learning intervention and experiment. 

In order to provide a first step towards this direction in video learning analytics, we present a 
video learning analytics system and the first results of the captured data. The system is open- 
source, web-based, and accessible to anyone who wants to design his own experimental design to 
perform experiments on student learning. The major practical implication of this study is the 
open-source video learning analytics system. Specifically, our empirical results indicate that the 
system is easy to use, helpful and applicable to any viewer, and smoothly incorporates any video 
lectures from YouTube and tests from Google Drive. The fact that the system is available for 
further improvement and experimentation makes its practical uses, implications, and 
perspectives even greater. 

This study produces additional implications for theory and practice. First, our study presents a 
clear outcome for instructors to engage video-assisted learning practices in their teaching. 
Students’ positive attitudes and slight increase in their performance proved benefits of providing 
video lectures throughout a course. In addition, the correspondence between the level of 
cognition/thinking each question required and the size of the respective peak (based on the 
“wisdom of the crowd”) provide implications for designers, developers, and video lecture creators. 
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Taking into account this correspondence, video-assisted learning platforms and video lectures 
can be developed by providing extra affordances like annotations, slower-pace, or even extra 
visual information to the students in these particular video segments. 

In our study, we investigated students’ video navigation and we explained how video production 
affects students’ learning performance in a video-assisted course. We also indicated the 
correspondence between the level of cognition/thinking each video segment requires and the size 
of the respective student activity peak. Last but not least, we presented students’ progress 
throughout the video-assisted course, and we examined students’ attitudes regarding easiness, 
usability, usefulness, and acceptance of the course. 

We want to emphasize that our findings are clearly preliminary with inevitable limitations. One 
important limitation is the small scale of the study (11 students); however, capturing and 
analyzing the experiences of eleven students over a long period of time allow us to understand 
how students use the respective materials. Furthermore, collecting repeated interactions 
minimizes the limitation of the small sample size. Our future research will concentrate on further 
refinement of the proposed system and analysis by applying and evaluating it on larger scale 
classes. This study can serve as a springboard for other scholars and practitioners to further 
examine the efficacy of video assignments and of this specific tool in particular. For those 
interested in using video-assisted learning approaches like flipping the classroom, this is an 
established flexible tool that can be used and adapted to meet their needs. 

Our future work will focus on collecting and triangulating analytics from more sources. By taking 
into account learners’ interactions and much other data, such as students’ demographic 
characteristics (gender, ethnicity, English-language skills, prior education and background 
knowledge, their success rate in each section, their emotional states, the speed at which they 
submit their answers, which video lectures seemed to help which students best in which sections, 
etc.), new avenues for significant research will open. Collecting the knowledge and experience of 
multiple learners will allow us to understand how students learn with the assistance of video 
lectures and then feed powerful algorithms to create seemingly personalized feedback. Future 
work will help to collect diverse data (i.e., success rate, emotional states), which will allow the 
community to consider the challenges for developing more personalized and effective video 
learning systems. These advancements will lead us to several interventions (e.g., to prevent drop¬ 
out), or to adaptive services and curricula. As the number of students grows and our ability to 
capture diverse analytics increases, we should expect even more exciting students-centered 
technologies to be explored and brought into learning. 
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